{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第三章的3件事\n",
    "\n",
    "    1.经典模型： 卷积神经网络\n",
    "    2.典型任务1：图像分类\n",
    "    3.典型任务2：目标检测\n",
    "    \n",
    "## 计算机视觉的因目的不同的分类\n",
    "\n",
    "    1. 给图像按人类的语义分类，\n",
    "        (a) Image Classification： 图像分类，用于识别图像中物体的类别（如：bottle、cup、cube）。\n",
    "        \n",
    "    2. 找到目标在图片中的位置\n",
    "        (b) Object Localization： 目标检测，用于检测图像中每个物体的类别，并准确标出它们的位置。\n",
    "\n",
    "    还不太理解，！！！！后期修改\n",
    "        (c) Semantic Segmentation： 图像语义分割，用于标出图像中每个像素点所属的类别，属于同一类别的像素点用一个颜色标识。\n",
    "\n",
    "    4.找到你，标注你： 精确抠图，不仅找到物体的位置还要将他们的轮廓标注出来\n",
    "        (d) Instance Segmentation： 实例分割，值得注意的是，（b）中的目标检测任务只需要标注出物体位置，而（d）中的实例分割任务不仅要标注出物体位置，还需要标注出物体的外形轮廓。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# 卷积神经网络\n",
    "\n",
    "**卷积神经网络** 是由几个概念放到一起合起来解决一个计算机视觉的方案总称；\n",
    "  \n",
    "  - 1.卷积（Convolution）\n",
    "  - 2.池化（Pooling）\n",
    "  - 3.激活函数ReLU\n",
    "  - 4.批归一化（Batch Normalization）\n",
    "  - 5.丢弃法（Dropout）:Dropout来防止过拟合\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## 卷积（Convolution）\n",
    "\n",
    "- 卷积核：\n",
    "\t>\t卷积原始图片得出输出图像数字矩阵的，基本单位。\n",
    "\n",
    "\n",
    "- 卷积计算 :\n",
    "\t>\t原始输入的图片经过卷积核的计算得出一个新的数字矩阵代表的图片。\n",
    "\n",
    "- 步幅（stride）\n",
    "\t>\t卷积核移动的跨度大小\n",
    "\n",
    "- 填充（padding）:\n",
    "\t>\t图片经过卷积核处理后，大小会变小所以填充一些0补充为原来的图片大小\n",
    "\n",
    "\n",
    "- 感受野（Receptive Field）\n",
    "\t>\t经过卷积核得出的图片，可以倒推对比原来的图片。当原来的图片发生变化，会导致卷积核得出的数字矩阵中对应区域的数字发生变化，从而发现原图片变了。 使计算机感受到图片的变化。感受野就是感受的区域隐喻人可以看到的视野的概念\n",
    "\n",
    "\n",
    "- 多输入通道、多输出通道和批量操作\n",
    "\t>\t因为图片可以是丰富多彩的，甚至是视频。单通道不足以表达真实的复杂场景，所以需要复杂的处理方式解决实际问题\n",
    "\n",
    "\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/87c96c0b407a44d68d738428862782490573fda24b29401c8cee29a4fc49074e\" width = \"700\"></center>\n",
    "\n",
    "\n",
    "    一个解释讲明白全部概念\n",
    "    首先看图 （a） 标注了一张图，\n",
    "    - 1-9 就是原始图分为9个区域\n",
    "    - 图中 2*2内容是0，1，2，3的方块名叫卷积核。\n",
    "    - 最右边的2*2的方块只有一个数字25的是经过卷积核处理后输出的图片数字矩阵。 \n",
    "    \n",
    "    故事： \n",
    "    \n",
    "    中间的卷积核就像是一块不能折叠的抹布要从左向右，从上向下的把原始图都擦一遍。这就是卷积核要做的事情。当它擦完以后，原始图发生了改变，首先缩水了，而且是有规律的缩水了。得出了输出图。\n",
    "    \n",
    "    - 这个规律是，首先把卷积核放在最开始的左上角， 此时最上角的数字*原始图最上角的数字 0*1，右上角的1*对应原始图同样位置的数字2一次相乘最后把数字相加 = 输出图片的数字矩阵25.\n",
    "    \n",
    "    - 卷积核移动的幅度就是步幅。 \n",
    "    \n",
    "    - 卷积核和原始图计算的过程就是卷积计算。\n",
    "    \n",
    "    - 输出的图片矩阵25是原始和卷积计算得来的，如果25这个数字发生了改变，首先卷积核是已知且固定的，那么就可以得出是原始图的数字发生了改变。这样变相的输出的图片数字就有了感受原始图变化的能力，这个能力好像是一个人专门盯着这个区域看。这就是感受野，或者视野范围能被感受的概念。\n",
    "    \n",
    "    -  当原始图片遇上不是1*1的卷积核，原始图片就注定缩水，但是缩水的时候需要不被别人发现。就把空白外围位置用0填充，虽然填充没有内容，但是大小保持了。\n",
    "    \n",
    "    -  一张二维平面的原始图能代表单色图片， 一张彩色图片就需要3原色组成3张二维平面合在一起才能表示，所以输入是多通道进入，处理完是多通道输出。如果是个视频，算最基本的一秒视频24张图片计算批量操作也是必须的。 最后多通道配个图吧\n",
    "\n",
    "<br></br>\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/b1087b021220484193c8f2cd42f9709440da963e1f524de4bf46cd806d5f6079\" width = \"800\"></center>\n",
    "<center><br>图10：多输入通道计算过程 </br></center>\n",
    "<br></br>\n",
    "    \n",
    "> 综上所述 卷积网络上的一切都是根据真实情况合情合理的出现了\n",
    "    \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.池化（Pooling）\n",
    "\n",
    ">\t特征映射(feature map)\n",
    "\n",
    "### 池化2中方式\n",
    "\n",
    "1.平均池化： 池化大小的窗口中的数据取平均数，作为整个区域的数字\n",
    "\n",
    "2.最大池化。池化大小的窗口中的数据取其中最大数，作为代表整个区域的数字\n",
    "\n",
    "### 池化的目的\n",
    "1. 降低模型对图片的变化的敏感度\n",
    "2. 使得图片体积变小，节省内存空间，增加运算速度\n",
    "\n",
    "### 池化相关参数\n",
    "1.窗口大小 \t\tpool_size=[kh, kw]\n",
    "2.窗口滑动步幅   pool_stride = [stride_h, stride_w]\n",
    "3.图片填充\t\tpadding = [ph, pw]\n",
    "4.常见参数配置\tkh = kw = 2, stride_h = stride_w =2 ph = pw = 0使得图片宽高减半\n",
    "\n",
    "### 池化的特点\n",
    "1. 没有学习参数\n",
    "2. 通道数不变， 每个通道独立的进行池化\n",
    "3. 对微小的位置变化具有鲁棒性(鲁棒性解释：**就是耐艹，强硬有力不被轻易影响**)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ReLU激活函数\n",
    "\n",
    "> 一句话现在主流用ReLU激活函数 ，怎么用不重要反正后面会讲\n",
    "\n",
    "之前用Sigmoid函数，但 因为Sigmoid函数在反向传播过程中，容易造成梯度的衰减，先用Relu激活函数\n",
    "\n",
    "\n",
    "下面的程序画出了Sigmoid和ReLU函数的曲线图："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAFFCAYAAAC393oCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd4VVW+//H3Cs2EEoTQi4AiCCgdKTKOoqio2NCfqHMRvaJjwUEQpl2VmXvnThBlwAGvKAKOgKNgwQaoYEuAEAi9CNIjHelICPn+/jgBE5oBzjnrlM/refZzcvbZOftzElj57rXX3suZGSIiIiISWRJ8BxARERGRE6lIExEREYlAKtJEREREIpCKNBEREZEIpCJNREREJAKpSBMRERGJQCrSRERERCKQijQRERGRCKQiTURERCQCFfcdIBhSUlKsTp06vmOISBjNnTt3u5lV8p3jXKn9Eok/RW2/YqJIq1OnDpmZmb5jiEgYOefW+c4QDGq/ROJPUdsvne4UERERiUAq0kREREQikIo0ERERkQikIk1EREQkAqlIExEREYlAKtJERM6Cc+5159xW59ziAusqOOc+c86tzH8832dGEYluKtJERM7OGOD649b9HvjCzOoDX+Q/FxE5K2Et0k525Hnc6845N8w5t8o5t9A51yKc+UREisrMvgZ2Hrf6FmBs/tdjgVvDGkpEYkq4e9LGcOKRZ0E3APXzl17Ay2HIJCISLFXMbFP+15uBKifbyDnXyzmX6ZzL3LZtW/jSiUhUCeuMA2b2tXOuzmk2uQV4w8wMmOWcK++cq1ag0RORs2EGubmQk/PLS24u5OWdfjH75W2O7reoj6VKQY8e4fuZhJiZmXPOTvHaSGAkQKtWrU66jYhEh8NHDjN09lDub3Y/KUkpQX3vSJsWqgawocDzjfnrTijSnHO9CPS2Ubt27bCEE/EuJwfWr4dNm2DbtsCydWvgcft22LMH9u6FffsCS8GvLcJrgfPPj4UibcvRA0vnXDVgq+9AIhJa7yx9h6c/e5qLK15M1wZdg/rekVakFZmORCVmmcHmzbBgASxcCEuXwpo1gSU7++deqoKSk6FSpcBjmTJQvXrgsWzZwGPp0pCYCCVKQMmSp1+KFQssCQknX5w79WsFt3EukK2ojwkxcR3TZKAH8Pf8xw/8xhGRUDIzBqUNolGlRtx08U1Bf/9IK9KygVoFntfMXycSu8xg8WL4+mv46iv45ptAkXZU9epQrx78+tdQt25gqVEjUJRVqgQpKYFThRJWzrkJwK+BFOfcRuBZAsXZ2865B4F1wF3+EopIqE39fioLtixg9C2jSXDBP9CMtCJtMvC4c+4t4HJgt8ajSUzKy4NZs2DiRJg0KXAKE6BWLejUCdq0gcsuCywVKvjNKidlZt1P8VKnsAYREW9S01KpUbYG91x6T0jeP6xF2imOPEsAmNn/AZ8AXYBVwAGgZzjziYTcli3w2mvwyiuwYUPg9OJ118Gzz8LVV8MFF/x8+k9ERCJWRnYGX679ksHXDqZksZIh2Ue4r+481ZHn0dcNeCxMcUTCZ8MG+NvfYNQoOHwYrrkG/vd/4eaboVw53+lEROQMpaalUv688vRq2Stk+4i0050isWXvXnjuOXjppcDzBx6APn2gQQOvsURE5Oyt2L6C95a9xx+u+ANlS5UN2X5UpImEyqRJ0Lt34HYZPXvCM88ETmeKiEhUe2HmC5QsVpLel/cO6X5i4pp3kYiyfz88+CB06wZVqsDMmYHTnCrQRESi3qa9mxi7YCw9m/WkSpmTTioSNOpJEwmmVauga1dYvhz+9KfAqc7i+m8mIhIrhs4eSm5eLn3b9w35vvTXQyRYZs+Gm24K3Pfss88Ct9IQEZGYsfun3byc+TJ3XHIHF1W4KOT70+lOkWD49FO46qrAHf9nzlSBJiISg16Z+wp7Du1hQIcBYdmfijSRczVjBtx2G1xyCaSnQ/36vhOJiEiQHco9xD9m/YNOdTvRsnrLsOxTpztFzsWsWYF7nV10EUybBhUr+k4kIiIh8ObCN9m0bxNjbx0btn2qJ03kbK1bFxiDVrVqYAyaCjQRiUBTp06lY8eORd7+7rvvZtSoUSFMFH3yLI/n05+nedXmXFPvmrDtV0WayNk4eBBuvz0we8Ann0C1ar4TiUiEqV69OomJiZQpU4bk5GSuv/56NmzYcMbvU7lyZf71r3+dsH7r1q0458jKyiq0ft26dTjnWLZsGWZGnz59GDhwYJH399xzz/HHP/6RgwcPnnHWWPXB8g9YsWMFAzoMwIVx6j4VaVGkc+fOXo9ufmn/Zka5cuWYPXs2EMNHb2bwyCMwbx6MGwcXX+w7kYhEmOzsbDZt2sS8efPYt28fq1evZuvWrQwYcGYDztetW8e2bdto1arVCa/NmTOHxMRELr300kLrMzIyKFu2LA0aNGDatGnk5ORw1VVXFXmfDRs25KKLLmLChAlnlDVWmRmpaanULV+XOxrdEdZ9q0iLEGlpaXTp0oVq1apRpkwZ6tWrx5///OdC20ybNo0HH3zQU8Jf3v+aNWs4cOAAl112WWwfvY0fD2+8EZgU/aabfKcRkbOQl5dHcnIyU6dOLbT+tttuo2/fc7//1Zw5cyhXrhwNGzYEoGLFirRo0YKtW7eesO20adO44oorqFChAnXq1GH06NHHXsvMzDxWcB0vMzOTZs2aUfy4ezFmZGTQsmVLEhISeP/997nmmmsK9f7885//5IILLuDQoUMALFu2jKpVq/LOO+8c2+baa6/l/fffP7cfQoz4et3XzM6eTb/2/SieEN6h/CrSIsCUKVO44YYb6NGjB+vWrWPnzp28/fbbNG/e3He0M7Jw4UIaNmxIYmJi7B69bdoETzwB7drBf/2X7zQicpYSEhK4/PLLycjIOLbu888/Z+bMmTz77LOFtn300UcpX778KZe///3vJ7z/nDlzaN26Nc45cnNzmTJlCv/+97+55557Cm03btw47rvvPgYOHMj27dt566236N27N0uXLj32Ps2bNych4cQ/15mZmbRu3fqE9RkZGcfWz5s3j0aNGhV6/ZFHHiExMZERI0awZs0aOnfuzP/+7/9y5513Htvm0ksvZd68eb/0Y4wLqWmpVEqqRM9mPcO/czOL+qVly5YWSkeOHLFy5crZlClTCq2/9dZb7amnnjrn97/pppvs7rvvPu0206dPt0qVKh17Pm7cOKtbt66VLl3aunXrZv369Sv0HtOnT7eKFSva8OHD7cILL7SkpCTr3bu3bdmyxbp3727JyclWp04d+/bbb499z4QJE6xp06ZWpkwZa9Cggb333nun3L+Z2b///W+78MILrUyZMnbXXXfZ73//e7vvvvvMzOyRRx6xhx9+uND2L730ktWuXdt++uknMzNbunSpValSxd5+++1j2zz77LN28803F/VHF155eWY332x23nlmy5f7ThP3gEyLgPbnXJdQt19yas8884zddNNNZmZ2+PBha9y4sY0aNSoo733ttddaqVKlLDk52YoVK2Zly5a1f//734W22bdvn51//vk2adKkQuuvvvpqe/31183MrFOnTta3b9+T7qNq1ar2r3/9q9C6I0eOWJkyZY61q/Xr17fRo0ef8L2TJ0+2ihUrWr169Wzo0KEnvD5t2jRLTEws8ueNVQs2LzCew/761V+D+r5Fbb/Uk1YEoT7iqlSpEp999hlDhw5l2bJlJ80wd+5cWrYM3Jdl1KhR/OEPf2D8+PHs3r2bjh07MnToUJo1a1Zo+z179nD48GGWLFnClClTGDZsGPfeey/9+vVjx44ddOzYkb/97W8AjBw5kj/84Q+8+uqr7N27lyFDhtC9e3fWr19/wv4BxowZQ79+/Rg3bhx79uzhiiuu4IUXXjjW+xeTR29vvQUffgj//d9wklMPIhJd2rdvf6xdHzFiBElJSfTsGZzekszMTCZMmMCuXbvYsGEDNWvWZPny5YW2+eqrr9i1axcPPPBAob8Ts2bNoly5cpgZc+fOPel4tOzsbDZv3lyoXQaYP38++/bto23btgCcf/757Nmz54Tvb9KkCfv37+eiiy6id+8TJwnfs2cPFSpUOJcfQUx4Pv15SpcozaOtH/UToCiVXKQv4TgSDeUR1549e+wvf/mLNW3a1JxzVrt27WNHUUfdfffd9qc//cn2799vFSpUsA8//PDYa/v37zfApk6dWmj7gj1re/fuNcCmT59+bN3gwYOtc+fOtmfPHqtQoUKh18zMatWqdewI7+j+j+7v+Aw//fRTofePuaO3ffvMatQwa9nSLDfXdxox9aTJudu1a5c552zOnDlWoUIFmzlz5km3e/jhh6106dKnXP7nf/6n0PYrV640wNauXXts3WuvvWbVqlWzI0eOHFs3duxYa9GixSnzfffddwbYypUrT3ht1qxZBtgPP/xQaH3fvn2tbdu2hbIff1Zj48aNVrduXXvyySetVKlStmzZshPe/7nnnjv2Ny9erf1xrRUbWMz6TOkT9PcuavvlvYEKxhKORm7KlClWuXJlMzMbOnSotW7d2vLy8oK+ny1btthvf/tbS0hIsA0bNhxbX79+fXv33Xft448/PpbjqHXr1hlgW7ZsKbT9W2+9dez5N998Y+XKlSvUQPTo0cP69etn77///gmnMvPy8iwxMdFmzJhRaP9mZp988omVK1eu0Odfs2aNAfbjjz+amVmbNm1OWoStXr3azjvvPOvcufNJP//EiROtRo0ap/0ZeTFwYOC/y9df+04i+VSkSTA0btzY6tWrZ//xH/8RtPccP368VahQodC6bdu2WUJCgn311VfH1i1ZssQSExPtgw8+sCNHjlhOTo4tWrTI5syZY2aBYS3Jycl28ODBQktubq7t3r3bkpOTrXfv3vbjjz/ajh077OWXX7bExET78ssvj+3j008/tYsuuujY861bt9oll1xiAwcONDOzBx980G688cYTPkOHDh3stddeC9rPJBr1/qS3Ff9LcVu/a33Q37uo7ZdOdxZR27Zt2bZtG5mZmQwcOJBhw4ad9F4pjzzyCGXKlDnlcvT04qlUrlyZbt26kZeXx4EDB4BAt/OqVato2bIlW7dupXz58oW+Z9KkSVSvXp3KlSsX2r7ggNKjpysLDj49um7Pnj2cf/75hd7zww8/JDExkbZt2xbaP8CWLVuoUqVKoc8/btw46tateyxb8+bNjw18PSo7O5tOnTrx8MMP89VXX53Q9Q+wePHiyLtgIjsbUlPhjjvgDG4pIiKRr127dmzbtu2kQ1HO1tHB/gWlpKTQvn17Jk6ceGxdo0aNGDVqFH/84x9JTk6mSpUqPPLIIxw+fBgInDLdvXs3iYmJhZa33nqLcuXK8dFHH5GZmUnNmjVp2LAh7777LlOmTOHKK688to/rrruO4sWL8+WXX7J7926uu+46unTpwjPPPAPAwIEDmT59Ol988cWx71mxYgUrV6484SKHeLLjwA5ey3qNey69h1rJtfwFKUolF+lLuI5EQ3HE9be//c2mTZtme/futSNHjtjcuXOtZcuWdt111x3bZsaMGZaSkmJmZpmZmZaQkGBffPGF5eTk2LvvvmvJycnWpUuXQtsffxR333332dNPP33s+f79+61YsWK2cuVKW7FihZUsWdK++uory83NtS+++MKqVat27HRlwf2bmWVkZFhCQoJNmzbNcnJy7J133rHSpUvb7bfffmybmDp669nTrGRJs++/951ECkA9aRIEnTp1stTUVN8xQurTTz+1jh07Fnn7u+++21599dUQJop8z814zngOW7xlcUjev6jtl/cGKhhLuBq5//zP/7SyZcueMAbgXPz5z3+2xo0bW9myZS05OdkuvfRSGzRokB08ePDYNkfHjh313HPPWYUKFSwlJcX69OljXbp0OVb8HN3+2muvLbSfSy65pNCVRWlpaZacnHzslOUbb7xh9erVs6SkJGvWrJm98847p9y/mdl//dd/WYUKFaxatWrWrVs3+9WvfmV//evPV7/k5eVZw4YNbcaMGbZr1y5r3rx5oSuUNm7caImJifb5558fW7d8+XKrXLmyHThw4Ix/jiGzapVZsWJmv/ud7yRyHBVpcq5eeeUVa9y4seXk5PiOIhFk36F9VjG1ot00PnRj8lSkhUAkHnHt3LnTypYtawsXLvQd5QQxcfT20EOBXrTsbN9J5Dgq0uRszZ4928qVK2dNmzY96aB5iW8vzX7JeA77Zt03IdtHUdsvF9g2urVq1coyMzNDuo+RI0cybNgwsrKyKFGiREj3dTpz5syhTJkyNGzYkO+//55evXpRs2ZN3njjDW+ZYtbGjVCvHvznf8KIEb7TyHGcc3PN7MR7E0SZcLRfIlI0uXm51H+pPtXLViftgbSQ7aeo7ZcuHPgFGRkZJCcnM2LECCZOnOi1QANYsmQJV111FWXKlOGGG26gY8eOjBw50mummDV4MOTlQf/+vpOIiEgYvL3kbdbuWsuADmc2x2qoqCdN5GS2b4fateGuu2DMGN9p5CTUkyYiwWRmNH+lOTlHclj86GISXOj6sYrafoV3plCRaDF6NBw8CP36+U4iIiJhMPX7qSzYsoDXu74e0gLtTERGCpFIkpcHL78Mv/oVNGniO42IiIRBaloqNcrW4N7L7vUd5RgVaSLHmzoV1qyBxx7znURERMIgIzuDL9d+SZ+2fShZrKTvOMeoSBM53ogRULUq3Hqr7yQiIhIGg9IGUf688vRq2ct3lEJUpIkUtHYtfPwxPPQQlIycoykREQmN73Z8x7vL3uXRVo9StlRZ33EKUZEmUtDrr4NzgSJNRERi3uD0wZQsVpLel/f2HeUEKtJEjjKDN9+ETp2glscJdUVEJCw27d3E2AVjub/Z/VQpU8V3nBOoSBM5aubMwAUD993nO4mIiITB0NlDyc3LpV/7yLzdkoo0kaPefBMSE+G223wnERGRENtzaA8vZ77MHZfcwUUVLvId56RUpIkA5OTA22/DLbdA2cgaOCoiIsH3SuYr7Dm0J2KmgDoZFWkiELg32o4dOtUpIhIHDuUeYsisIXSq24mW1Vv6jnNKmhZKBAKnOlNSoHNn30lERCTE3lz4Jpv2bWLsrWN9Rzkt9aSJ/PRT4N5ot98OJUr4TiMiIiGUZ3k8n/48zas255p61/iOc1rqSRP54gvYv18XDIiIxIEPln/Aih0rmHDHBJxzvuOclnrSRN57D8qVg6uv9p1EYoRzro9zbolzbrFzboJz7jzfmUQEzIzUtFTqlq9Lt0bdfMf5RSrSJL4dOQKTJ0OXLpoGSoLCOVcD6A20MrMmQDHgbr+pRATgm/XfMDt7Nv3a96N4QuSfTAx7keacu945t8I5t8o59/uTvF7bOTfDOZflnFvonOsS7owSR9LTYds2neqUYCsOJDrnigNJwA+e84gIkJqWSqWkSvRs1tN3lCIJa5HmnCsGDAduABoB3Z1zjY7b7M/A22bWnMDR54hwZpQ48/77gR6066/3nURihJllA4OB9cAmYLeZTfObSkQWbVnEJys/offlvUkskeg7TpGEuyetDbDKzFabWQ7wFnDLcdsYUC7/62R0BCqhYhYYj3bNNYExaSJB4Jw7n0C7VheoDpR2zt133Da9nHOZzrnMbdu2+YgpEncGpQ+idInSPNr6Ud9RiizcRVoNYEOB5xvz1xX0HHCfc24j8AnwRHiiSdxZtiwwV2fXrr6TSGy5BlhjZtvM7DDwLtC+4AZmNtLMWplZq0qVKnkJKRJP1u1ax4RFE3ioxUNUSKzgO06RReKFA92BMWZWE+gC/Ms5d0JOHYnKOZs6NfB4ww1+c0isWQ+0dc4lucD1/Z2AZZ4zicS1F2e+iHOOp9o95TvKGQl3kZYN1CrwvGb+uoIeBN4GMLOZwHlAyvFvpCNROWdTp0LDhlC7tu8kEkPMbDYwEZgHLCLQzo70Gkokju04sIPXsl7jnkvvoVZyrV/+hggS7iJtDlDfOVfXOVeSwIUBk4/bZj2BI0+cc5cQKNLUVSbBdfAgfPWVLhiQkDCzZ82soZk1MbPfmNkh35lE4tXwOcM5cPgA/dv39x3ljIW1SDOzXOBxYCqB7v+3zWyJc+4vzrmjA4P6Ag855xYAE4D7zczCmVPiwNdfB6aDuu4630lERCREDhw+wEsZL3HTxTfRuHJj33HOWNjv5GZmnxC4IKDgumcKfL0U6BDuXBJnpk6FUqXgV7/ynURERELk9azX2X5gOwM6DPAd5axE4oUDIqE3ZQpceSUkJflOIiIiIZCbl8sLM1+gXc12dKgVnX0/KtIk/mzYELj9hk51iojErLeXvM3aXWsZ0GFAxE+kfioq0iT+TMu/+buKNBGRmGRmDEobxCUpl3Bzg5t9xzlrkT+7qEiwffklVKkCjY6fkUxERGLBtO+nsWDLAl7v+joJJ95qNWpEb3KRs2EWKNJ+/WuI0u5vERE5vdS0VGqUrcG9l93rO8o5UZEm8WX1ati4MVCkiYhIzJmTPYcZa2fQp20fShYr6TvOOVGRJvFlxozAo4o0EZGYlJqWSnKpZHq17OU7yjlTkSbx5eh4tAYNfCcREZEg+27Hd7y77F0ebf0oZUuV9R3nnKlIk/ih8WgiIjFtcPpgShYryZOXP+k7SlCoSJP48f33kJ2tU50iIjFo877NjF0wlvub3U+VMlV8xwkKFWkSP778MvCoIk1EJOYMnTWU3Lxc+rXv5ztK0KhIk/ih8WgiIjFpz6E9vJz5MndccgcXVbjId5ygUZEm8ePbb6FjR41HExGJMa9kvsLuQ7vp36G/7yhBpSJN4kN2NqxbB+3b+04iIiJBdCj3EENmDeHqulfTqnor33GCStNCSXxITw88dujgN4eIiATVmwvfZNO+TYy5dYzvKEGnnjSJD+npcN550KyZ7yQiIhIkeZbH8+nP07xqc66td63vOEGnnjSJD+np0KYNlIzuKUJERORnk1dMZsWOFUy4YwIuBscbqydNYt+BAzBvnsajiYjEEDMjNS2VuuXr0q1RN99xQkI9aRL7MjMhN1fj0UREYsg3679h1sZZDO8ynOIJsVnOqCdNYl9aWuCxXTu/OUREJGhS01KplFSJns16+o4SMirSJPalp0PDhlCxou8kIiISBIu2LOKTlZ/wRJsnSCyR6DtOyKhIk9hmFijSNB5NRCRmPJ/+PKVLlOaxNo/5jhJSKtIktq1aBTt3Qtu2vpOIiEgQrN+9ngmLJ/BQi4eokFjBd5yQUpEmsS0jI/B4+eV+c4iISFC8OPNFAJ5q95TnJKGnIk1iW0YGlC4NjRv7TiIiIudox4EdvDrvVe659B5qJdfyHSfkVKRJbMvIgJYtoVgx30lEROQcDZ8znAOHD/B0+6d9RwkLFWkSu3JyICsrMNOAiIhEtQOHD/BSxkvcWP9GmlRu4jtOWKhIk9i1cCEcOqQiTUQkBrye9TrbD2xnQIcBvqOEjYo0iV1HLxpQkSYiEtVy83J5YeYLtKvZjitqX+E7TtjE5jwKIhAo0ipXhtq1fScREZFz8M6Sd1i7ay3/uO4fMTmR+qmoJ01iV0ZGoBctjv5Di4jEmqMTqV+Scgk3N7jZd5ywUpEmsWn3bli+XKc6RUSi3LTvp7FgywKebv80CS6+ypb4+rQSP+bODUwJpSJNRCSqpaalUr1sde697F7fUcJORZrEprlzA48tW/rNISIiZ21O9hxmrJ1Bn7Z9KFmspO84YaciTWJTVhbUqgUpKb6TiIjIWRqUPojkUsn0atnLdxQvVKRJbJo3D5o3951CRETO0sodK5m0dBKPtn6UcqXK+Y7jhYo0iT379sF336lIExGJYoPTB1OyWEmevPxJ31G8UZEmsWfhwsBFAyrSRESi0uZ9mxm7YCz3N7ufKmWq+I7jjYo0iT1ZWYFHFWkiIlFp6Kyh5BzJoV/7fr6jeKUiTWJPVhZUqBC4cEDEA+dceefcROfccufcMudcO9+ZRKLFnkN7eDnzZe5odAcXVbjIdxyvNC2UxJ6srEAvmmYaEH+GAlPMrJtzriSQ5DuQSLR4JfMVdh/aHVcTqZ9K2HvSnHPXO+dWOOdWOed+f4pt7nLOLXXOLXHOjQ93Rolihw/D4sXQooXvJBKnnHPJwK+AUQBmlmNmu/ymEokOh3IP8Y/Z/+DqulfTqnor33G8C2tPmnOuGDAcuBbYCMxxzk02s6UFtqkP/AHoYGY/OucqhzOjRLmlSyEnR+PRxKe6wDZgtHOuKTAXeNLM9vuNJRL5xi0axw97f2D0LaN9R4kI4e5JawOsMrPVZpYDvAXcctw2DwHDzexHADPbGuaMEs100YD4VxxoAbxsZs2B/UChswbOuV7OuUznXOa2bdt8ZBSJOHmWx6C0QTSv2pxr613rO05ECHeRVgPYUOD5xvx1BV0MXOycS3POzXLOXX+yN1IjJyeVlQVJSVC/vu8kEr82AhvNbHb+84kEirZjzGykmbUys1aVKlUKe0CRSDR5xWRW7FhB/w79cRpTDETm1Z3FgfrAr4HuwKvOufLHb6RGTk4qKwuaNoVixXwnkThlZpuBDc65BvmrOgFLT/MtInHPzEhNS6Vu+bp0a9TNd5yIEe4iLRsoeF+EmvnrCtoITDazw2a2BviOQNEmcnp5eTB/vk51SiR4AhjnnFsINAP+5jmPSET7Zv03zNo4i77t+lI8QTeeOCrcP4k5QH3nXF0CxdndwD3HbfM+gR600c65FAKnP1eHNaVEp9WrYe9eFWninZnNB3RpmkgRDUobREpSCj2b9/QdJaKEtSfNzHKBx4GpwDLgbTNb4pz7i3Oua/5mU4EdzrmlwAzgaTPbEc6cEqV00YCISNRZvHUxH6/8mN5tepNUQrcULCjsfYpm9gnwyXHrninwtQFP5S8iRZeVBcWLQ5MmvpOIiEgRDUobROkSpXmszWO+o0ScSLxwQOTszJsHjRpBqVK+k4iISBGs372eCYsn8FCLh6iQWMF3nIijIk1ig9nP00GJiEhUeHHmiwD0adfHc5LIpCJNYsOmTbB1q4o0EZEosePADl6d9yrdm3SndnJt33Eikoo0iQ26aEBEJKqMmDOCA4cP0L9Df9+zqZTsAAAc90lEQVRRIpaKNIkNR4u0Zs385hARkV904PABhmUM48b6N9Kksi72OhUVaRIbsrLgoougXDnfSURE5BeMzhrN9gPbGdBhgO8oEU1FmsQGXTQgIhIVcvNyGTxzMO1qtuOK2lf4jhPRVKRJ9Nu1C9asUZEmIhIF3lnyDmt3rWVAhwGaSP0XqEiT6Dd/fuBRRZqISEQ7OpF6w5SG3NzgZt9xIp5mMZXopys7RUSiwrTvp7FgywJGdR1FglM/0S/RT0iiX1YWVKsGVar4TiIiIqcxKH0Q1ctW595L7/UdJSqoSJPop4sGREQiXuYPmUxfM50+bftQqrim7ysKFWkS3Q4ehGXLVKSJiES41LRUkksl06tlL99RooaKNIluixfDkSMq0kREItjKHSuZtHQSj7Z+lHKldD/LolKRJtFNFw2IiES8wemDKVmsJL0v7+07SlRRkSbRbd48SE6GunV9JxERkZPYvG8zYxeMpUfTHlQtU9V3nKhS5CLNOXezc7peViJMVlZgvk7dEFGKSG2ZSHgNmz2MnCM59Gvfz3eUqHMmDdX7wEbnXKpz7pJQBRIpstxcWLgQWrTwnUSii9oykTDZc2gPI+aM4I5Gd1C/Yn3fcaLOmRRpFwKvAncBi51zM51zDznnNAJQ/FixAn76SePR5EypLRMJk5FzR7L70G5NpH6WilykmdlaM3vWzOoC1wKrgCHAJufcv5xzV4UqpMhJ6aIBOQtqy0TC41DuIYbMGsLVda+mVfVWvuNEpbMal2Fm083sN8DFwFzgXuBz59xq51wf55ymm5LQy8qC886Dhg19J5EopbZMJHTGLRrHD3t/UC/aOTirIs05d6VzbgywAmgCDAc6AxOBgcAbwQoockpZWXDppVBcf0fl7KgtEwmNPMtjUNogmlVtxrX1rvUdJ2oV+a+bc+4CoEf+Ugf4EugFvGtmh/I3+8I5NxN4M7gxRY5jFijS7rrLdxKJMmrLRELvwxUfsmLHCsbfPh6nq+/P2pl0QawGfgDGAK+b2ZpTbLcEyDjHXCKnt24d7Nql8WhyNtSWiYSQmZGalkrd8nW5s/GdvuNEtTMp0m4CpppZ3uk2MrPvAA28ldDSRQNy9tSWiYTQt+u/ZebGmfzzhn9SPEHDUc5FkX96ZvZpKIOInJGsLEhICIxJEzkDastEQis1LZWUpBR6Nu/pO0rU0123JTplZQWu6kxK8p1ERETyLd66mI9XfkzvNr1JKqH2+VypSJPolJWlU50iIhFmUNogkkok8Vibx3xHiQkq0iT6bNsG2dkq0kREIsj63euZsHgCD7V4iAqJFXzHiQkq0iT66KIBEZGIM2TmEACeaveU5ySxQ0WaRJ958wKPKtJERCLCzoM7eXXeq3Rv0p3aybV9x4kZKtIk+mRlQZ06cP75vpOIiAgwPGM4+w/vp3+H/r6jxBQVaRJ9dNGAiEjEOHD4AMMyhnFj/RtpUrmJ7zgxRUWaRJe9e2HlShVpIiIRYnTWaLYf2K6J1ENARZpElwULAo8q0kREvMvNy2XwzMG0q9mOK2pf4TtOzFGRJtFFV3aKiESMiUsnsnbXWvp36K+J1ENARZpEl6wsqFQJqlf3nUREJK4dnUi9YUpDujbo6jtOTNLMpxJdjl40oCM2ERGvPlv9GfM3z2dU11EkOPX5hIJ+qhI9fvoJFi+Gli19JxERiXupaalUL1udey+913eUmKUiTaLHokWQm6siTaKCc66Ycy7LOfeR7ywiwZb5QybT10ynT9s+lCpeynecmBX2Is05d71zboVzbpVz7ven2e4O55w551qFM59EsKMzDbRo4TeHSNE8CSzzHUIkFFLTUkkulUyvlr18R4lpYS3SnHPFgOHADUAjoLtzrtFJtitLoIGbHc58EuHmzg3MMlCnju8kIqflnKsJ3Ai85juLSLCt2rmKSUsn8dtWv6VcqXK+48S0cPektQFWmdlqM8sB3gJuOcl2fwVSgZ/CGU4i3Lx5gV40XTQgke8fQH8gz3cQkWAbnD6YksVK8mTbJ31HiXnhLtJqABsKPN+Yv+4Y51wLoJaZfRzOYBLhcnICY9I0Hk0inHPuJmCrmc09zTa9nHOZzrnMbdu2hTGdyLnZvG8zY+aPoUfTHlQtU9V3nJgXURcOOOcSgBeBvkXYVo1cPFm8OFCoqUiTyNcB6OqcW0vgbMHVzrk3C25gZiPNrJWZtapUqZKPjCJnZdjsYeQcyaFf+36+o8SFcBdp2UCtAs9r5q87qizQBPgyv4FrC0w+2cUDauTijC4akChhZn8ws5pmVge4G5huZvd5jiVyzvYc2sOIOSO4o9Ed1K9Y33ecuBDuIm0OUN85V9c5V5JAAzb56ItmttvMUsysTn4DNwvoamaZYc4pkWbuXEhOhgsv9J1ERCQujZw7kt2Hdmsi9TAKa5FmZrnA48BUApemv21mS5xzf3HOaU4JObV58zTTgEQdM/vSzG7ynUPkXB3KPcSQWUO4uu7VtKquO2OFS9inhTKzT4BPjlv3zCm2/XU4MkmEO3wYFiyAxx/3nUREJC6NXzSeH/b+wOtdX/cdJa5E1IUDIie1dCkcOqSLBkREPMizPAalD6JZ1WZ0vrCz7zhxRROsS+TTRQMiIt58uOJDlm9fzvjbx+M05CSs1JMmkW/uXChTBurraiIRkXAyM1LTUqlbvi53Nr7Td5y4oyJNIt/RiwYS9M9VRCScvl3/LTM3zqRvu74UT9DJt3DTXz2JbLm5MH++xqOJiHiQmpZKSlIKPZv39B0lLqlIk8i2fDkcPKgiTUQkzBZvXczHKz/miTZPkFQiyXecuKQiTSLbnDmBRxVpIiJh9Xz68ySVSOKx1o/5jhK3VKRJZMvIgHLloEED30lEROLG+t3rGb9oPA+1eIiKSRV9x4lbKtIksmVkQOvWumhARCSMhswcAsBT7Z7ynCS+6S+fRK6DB2HhQmjTxncSEZG4sfPgTl6d9yrdm3SndnJt33Himoo0iVzz5weu7rz8ct9JRETixvCM4ew/vJ/+Hfr7jhL3VKRJ5Jo9O/ConjQRkbA4ePggwzKGcWP9G2lSuYnvOHFPRZpErowMqFkTqlXznUREJC6Mnj+a7Qe2qxctQqhIk8iVkaFeNBGRMMnNy2Vw+mDa1mxLx9odfccRVKRJpNqxA77/XkWaiEiYTFw6kTW71jCgwwBNpB4hVKRJZDp6E1sVaSIiIXd0IvWGKQ3p2qCr7ziST7OlSmSaPRuc00wDIiJh8Nnqz5i/eT6juo4iwan/JlLoNyGRaeZMaNw4MNuAiIiEVGpaKtXLVufeS+/1HUUKUJEmkefIkUCR1qGD7yQiIjEv84dMpq+Zzu8u/x2lipfyHUcKUJEmkWfpUtizB9q3951ERCTmDUobRHKpZB5u9bDvKHIcFWkSedLTA4/qSRMRCalVO1cxadkkftvqt5QrpeElkUZFmkSetDSoXBnq1fOdREQkpg1OH0yJhBI82fZJ31HkJFSkSeRJTw/0ouk+PSIiIbN532bGzB9Dj6Y9qFqmqu84chIq0iSybNkSuImtxqOJiITUsNnDyDmSQ7/2/XxHkVNQkSaRRePRRERCbu+hvYyYM4I7Gt1B/Yr1fceRU1CRJpElPR1KlYIWLXwnERGJWSPnjmT3od30b6+J1COZijSJLGlp0KpVoFATEZGgyzmSw5BZQ7iqzlW0rtHadxw5DRVpEjn274fMTLjiCt9JRERi1riF48jem82ADgN8R5FfoCJNIsfMmXD4MFx1le8kIiIxKc/yGJQ+iGZVm9H5ws6+48gv0ATrEjm+/BKKFdNFAyIiIfLhig9Zvn05428fj9NtjiKeetIkcnz5JbRuDWXK+E4iIhKTBqUPok75OtzZ+E7fUaQIVKRJZNi/HzIy4Ne/9p1ERCQmfbv+W9I3pNO3XV+KJ+hEWjRQkSaR4eh4NBVpIiIhkZqWSkpSCg80f8B3FCkiFWkSGTQeTUQkZBZvXcxH333EE22eIKlEku84UkQq0iQyaDyaiEjIPJ/+PEklknis9WO+o8gZUJEm/mk8mohIyKzfvZ7xi8bzUIuHqJhU0XccOQMq0sS/b77ReDQRkRAZMnMIAE+1e8pzEjlTKtLEv6lT4bzz4Fe/8p1ERCSm7Dy4k1fnvUr3Jt2pnVzbdxw5QyrSxL8pU+DKKyEx0XcSEZGYMmLOCPYf3s/T7Z/2HUXOgoo08Wv9eli+HK67zncSEZGYcvDwQYbNHkaX+l24tMqlvuPIWVCRJn5NnRp4VJEmMcI5V8s5N8M5t9Q5t8Q596TvTBKfRs8fzbYD2zSRehQLe5HmnLveObfCObfKOff7k7z+VH7jttA594Vz7oJwZ5QwmjIFatWCSy7xnUQkWHKBvmbWCGgLPOaca+Q5k8SZ3LxcBqcPpm3NtnSs3dF3HDlLYS3SnHPFgOHADUAjoPtJGq8soJWZXQZMBAaFM6OE0eHD8PnngV40TfQrMcLMNpnZvPyv9wLLgBp+U0m8mbh0Imt2rWFAhwGaSD2KhbsnrQ2wysxWm1kO8BZwS8ENzGyGmR3IfzoLqBnmjBIus2fDnj061SkxyzlXB2gOzPabROKJmTEobRANKjaga4OuvuPIOQh3kVYD2FDg+UZOf4T5IPDpyV5wzvVyzmU65zK3bdsWxIgSNp9+GpgKqlMn30lEgs45VwaYBPzOzPYc95raLwmZz1d/TtbmLJ5u/zQJTkPPo1nE/vacc/cBrYDnT/a6mY00s1Zm1qpSpUrhDSfB8cEHgXujnX++7yQiQeWcK0GgQBtnZu8e/7raLwml1LRUqpetzn2X3ec7ipyjcBdp2UCtAs9r5q8rxDl3DfAnoKuZHQpTNgmnlSthyRK49VbfSUSCygUGAI0ClpnZi77zSHyZ+8NcvljzBb+7/HeUKl7Kdxw5R+Eu0uYA9Z1zdZ1zJYG7gckFN3DONQdeIVCgbQ1zPgmX998PPKpIk9jTAfgNcLVzbn7+0sV3KIkPqWmpJJdK5uFWD/uOIkFQPJw7M7Nc59zjwFSgGPC6mS1xzv0FyDSzyQROb5YB3sm/ImW9mWnkY6x57z1o0QJqa5oSiS1m9i2gy+kk7FbtXMWkZZPo374/5UqV8x1HgiCsRRqAmX0CfHLcumcKfH1NuDNJmG3aBLNmwcCBvpOIiMSMF9JfoERCCZ5sq/snx4qIvXBAYtjkyWAGt93mO4mISEzYsm8Lo+ePpkfTHlQtU9V3HAkSFWkSfu+9BxdeCI0b+04iIhIThs0eRs6RHPq27+s7igSRijQJr23bArMMdOumWQZERIJg76G9jMgcwe2X3M7FFS/2HUeCSEWahNe//w1HjsB9un+PiEgwjJw7kl0/7dJE6jFIRZqE15tvQtOm0KSJ7yQiIlEv50gOQ2YN4ao6V9G6RmvfcSTIVKRJ+KxcGZivU71oIiJBMW7hOLL3ZqsXLUapSJPwGTcuMA6te3ffSUREol6e5fF8+vM0rdKUzhd29h1HQiDs90mTOGUWKNKuvhpq1PCdRkQk6n303Ucs276McbePw+lCrJiknjQJj2+/hVWrdKpTRCRIUtNSqVO+Dnc1vst3FAkRFWkSHiNGQHIy3KXGRETkXH27/lvSN6TTt11fiifopFisUpEmobd5M0yaBD17QlKS7zQiIlEvNS2VlKQUHmj+gO8oEkIq0iT0Ro2Cw4fhkUd8JxERiXqLty7mo+8+4ok2T5BUQge+sUxFmoRWbi688gpccw00aOA7jYhI1BucPpikEkk81vox31EkxFSkSWh99BFs2ACPPuo7iYhI1NuwewPjFo3joRYPUTGpou84EmIq0iR0zODvf4cLLoCbb/adRkQk6g2ZNQQzo0/bPr6jSBjokhAJnenTAzMMvPwyFNc/NRGRc7Hz4E5Gzh1J90u7c0H5C3zHkTBQT5qEzv/8D1SrBvff7zuJiEjUGzFnBPsP76d/+/6+o0iYqHtDQiM9HWbMgBdfhPPO851GRCSqHTx8kGGzh9GlfhcurXKp7zgSJupJk9D4y18gJQV69fKdREQk6o2ZP4ZtB7ZpIvU4o540Cb6pUwPL4MFQurTvNCIiUS03L5fBMwfTtmZbOtbu6DuOhJF60iS4cnOhb1+48EJ4/HHfaUREot6kpZNY/eNqBnQYoInU44x60iS4XnsNliwJTANVqpTvNCIiUc3MSE1LpUHFBnRt0NV3HAkzFWkSPDt3wjPPwJVXwm23+U4jIhL1Pl/9OVmbs3jt5tdIcDr5FW/0G5fg6d0bfvwR/vEPUJe8iMg5S01LpXrZ6tx32X2+o4gHKtIkOD74AMaNgz/9CZo1851GRCTqzf1hLl+s+YLfXf47ShXX8JF4pCJNzt2OHfDww9C0Kfzxj77TiIjEhEHpg0gulczDrR72HUU80Zg0OTdHjsBvfhMYjzZlCpQs6TuRiEjU+37n90xcOpH+7ftTrlQ533HEExVpcm6eew4+/TQwP6dOc4qIBMXg9MEUTyhO78t7+44iHul0p5y999+H//5veOCBwOlOERE5Z1v2bWH0/NH0aNqDamWr+Y4jHqlIk7Pz1VfQvTu0bg3Dh+tqThGRIBk2exg5R3Lo176f7yjimYo0OXNz5sDNN0PduvDxx5pAXUQkSPYe2suIzBHcfsntXFzxYt9xxDMVaXJm0tLguusCk6d/9hlUquQ7kYhIzHh13qvs+mmXJlIXQEWanImJE6FTp0Bh9sUXUKOG70QiIjEj50gOL858kavqXEXrGq19x5EIoCJNfllubuAqzrvugpYtIT09cKpTRESCZvyi8WTvzVYvmhyjW3DI6W3YAPfdB19/Df/xH/B//weJib5TiYjElDzLY1DaIJpWaUrnCzv7jiMRQj1pcnKHD8MLL0CjRjB3LrzxBowdqwJNRCQEPvruI5ZtX0b/Dv1xulpe8qlIk8Ly8gJjz5o1g3794MorYeHCwKwCIiISEqlpqdQpX4e7Gt/lO4pEEBVpEnDwIIwZA82bw513BqZ7eu89+PBDqFfPdzoRkZiVtj6N9A3p9G3Xl+IJGoUkP9O/hnh25Ah88w1MmgTjxwfm32zUCN58E+6+G4oV851QRCTmpaalkpKUwgPNH/AdRSKMirR4k50duAhg+nSYPBm2bg3cjLZrV/jtbwOnNzUeQkQkLJZsXcKH333IwF8PJKlEku84EmHCXqQ5564HhgLFgNfM7O/HvV4KeANoCewA/p+ZrQ13zqiXlwebNsGiRbBgQWDJyIDvvw+8XrYs3HADdOsWeCxTxm9ekRjyS+2cyFHPpz9PUokkHmv9mO8oEoHCWqQ554oBw4FrgY3AHOfcZDNbWmCzB4Efzewi59zdQCrw/8KZM6IdOQJ79sC2bT8vW7cGHrOzYc2awLJ2LRw69PP31a4dGG/26KOB3rKmTaG4OlJFgq2I7ZwIG3ZvYNyicTza6lEqJlX0HUciULj/SrcBVpnZagDn3FvALUDBxusW4Ln8rycC/3TOOTOzc977vn2BO+WbFV7y8sKzLjcXcnICy+HDp//6p58Cefftg717f/76wIFTf77zzw/cZLZJk8Dpy7p1A2PMLrss8JqIhENR2rmz8s26b/jxpx/P9W0kQry95G3MjKfaPeU7ikSocBdpNYANBZ5vBC4/1TZmluuc2w1UBLYX3Mg51wvoBVC7du2i7f2HH+DWW88md3A5ByVLQokSgcfjvz66lC0LFSsGTkWWLRt4LFMGypULTM1UuXLg8eiiic5FIsEvtnNn1X4BT3/2NLOzZwchokSKHk17cEH5C3zHkAgVtee7zGwkMBKgVatWRetlq107cGNW535eEhIKPw/luhIlAkuxYhqcLxLHzqr9AsbcOoYDh0/Tmy5RxeG4pNIlvmNIBAt3kZYN1CrwvGb+upNts9E5VxxIJnABwbk77zxo0SIobyUicgpFaefOSsOUhsF4GxGJEuG+me0coL5zrq5zriRwNzD5uG0mAz3yv+4GTA/KeDQRkfAoSjsnIvKLwtqTlj/G7HFgKoFL0183syXOub8AmWY2GRgF/Ms5twrYSaCBExGJCqdq5zzHEpEoFPYxaWb2CfDJceueKfD1T8Cd4c4lIhIsJ2vnRETOlObuFBEREYlAKtJEREREIpCKNBEREZEIpCJNREREJAKpSBMRERGJQCrSRERERCKQijQRERGRCORi4Wb+zrltwLoz+JYUjpuwPUbFw+eMh88I+pwnc4GZVQplmHBQ+3VK+pyxRZ+zsCK1XzFRpJ0p51ymmbXynSPU4uFzxsNnBH1O+Vm8/Iz0OWOLPufZ0elOERERkQikIk1EREQkAsVrkTbSd4AwiYfPGQ+fEfQ55Wfx8jPS54wt+pxnIS7HpImIiIhEunjtSRMRERGJaHFTpDnn7nTOLXHO5TnnWh332h+cc6uccyucc9f5yhhszrnnnHPZzrn5+UsX35mCyTl3ff7vbJVz7ve+84SKc26tc25R/u8w03eeYHHOve6c2+qcW1xgXQXn3GfOuZX5j+f7zBhJ1IbFVhum9iv6haMNi5siDVgM3A58XXClc64RcDfQGLgeGOGcKxb+eCEzxMya5S+f+A4TLPm/o+HADUAjoHv+7zJWXZX/O4ylS9jHEPg/V9DvgS/MrD7wRf5zCVAbFiNtmNqvmDGGELdhcVOkmdkyM1txkpduAd4ys0NmtgZYBbQJbzo5C22AVWa22sxygLcI/C4lSpjZ18DO41bfAozN/3oscGtYQ0UwtWExRe1XDAhHGxY3Rdpp1AA2FHi+MX9drHjcObcwv1s2lk4dxfrvrSADpjnn5jrnevkOE2JVzGxT/tebgSo+w0SJWP+/EIttWKz/zgqKp/YLgtyGFT/3PJHDOfc5UPUkL/3JzD4Id55wON1nBl4G/krgP8lfgReAB8KXToLkCjPLds5VBj5zzi3PP4KLaWZmzrm4uvxcbVghasNiQ1y2XxCcNiymijQzu+Ysvi0bqFXgec38dVGhqJ/ZOfcq8FGI44RTVP/ezoSZZec/bnXOvUfgVEmsNnJbnHPVzGyTc64asNV3oHBSG3ZqMdaGRfXv7EzEWfsFQW7DdLoTJgN3O+dKOefqAvWBDM+ZgiL/H8hRtxEYeBwr5gD1nXN1nXMlCQycnuw5U9A550o758oe/RroTGz9Ho83GeiR/3UPICZ7j4JMbVj0UfsVu4LahsVUT9rpOOduA14CKgEfO+fmm9l1ZrbEOfc2sBTIBR4zsyM+swbRIOdcMwKnCtYCD/uNEzxmluucexyYChQDXjezJZ5jhUIV4D3nHAT+v443syl+IwWHc24C8GsgxTm3EXgW+DvwtnPuQWAdcJe/hJFFbVjstGFqv2JDONowzTggIiIiEoF0ulNEREQkAqlIExEREYlAKtJEREREIpCKNBEREZEIpCJNREREJAKpSBMRERGJQCrSRERERCKQijQRERGRCKQiTaKKc668c26jc+6N49ZPds5955xL8pVNROR01H7JmVKRJlHFzHYBDwK/cc7dAuCc6wncCPQwswM+84mInIraLzlTmhZKopJz7hXgVuB6YAbwipkN8JtKROSXqf2SolKRJlHJOVcGWAhUB1YBLc3skN9UIiK/TO2XFJVOd0pUMrN9wEdAKWCUGjgRiRZqv6So1JMmUck51xpIBxYBFwCNzWyz31QiIr9M7ZcUlYo0iTrOufOAecBq4C5gAbDMzLp6DSYi8gvUfsmZ0OlOiUb/DVQFHsq/Gup+4Ebn3P0+Q4mIFIHaLyky9aRJVHHOdQC+Bn5jZuMLrH8eeAhoYmYbfeUTETkVtV9yplSkiYiIiEQgne4UERERiUAq0kREREQikIo0ERERkQikIk1EREQkAqlIExEREYlAKtJEREREIpCKNBEREZEIpCJNREREJAKpSBMRERGJQP8f3/15vVC5+00AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ReLU和Sigmoid激活函数示意图\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as patches\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "\n",
    "# 创建数据x\n",
    "x = np.arange(-10, 10, 0.1)\n",
    "\n",
    "# 计算Sigmoid函数\n",
    "s = 1.0 / (1 + np.exp(0. - x))\n",
    "\n",
    "# 计算ReLU函数\n",
    "y = np.clip(x, a_min=0., a_max=None)\n",
    "\n",
    "#####################################\n",
    "# 以下部分为画图代码\n",
    "# Sigmoid\n",
    "f = plt.subplot(121)\n",
    "plt.plot(x, s, color='r')\n",
    "currentAxis=plt.gca()\n",
    "plt.text(-9.0, 0.9, r'$y=Sigmoid(x)$', fontsize=13)\n",
    "currentAxis.xaxis.set_label_text('x', fontsize=15)\n",
    "currentAxis.yaxis.set_label_text('y', fontsize=15)\n",
    "\n",
    "# ReLU\n",
    "f = plt.subplot(122)\n",
    "plt.plot(x, y, color='g')\n",
    "plt.text(-3.0, 9, r'$y=ReLU(x)$', fontsize=13)\n",
    "currentAxis=plt.gca()\n",
    "currentAxis.xaxis.set_label_text('x', fontsize=15)\n",
    "currentAxis.yaxis.set_label_text('y', fontsize=15)\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 批归一化（Batch Normalization）\n",
    "\n",
    "[批归一化方法方法](https://arxiv.org/abs/1502.03167)（Batch Normalization，BatchNorm）\n",
    "\n",
    "BatchNorm能够使神经网络中间层的输出变得更加稳定，并有如下三个优点：\n",
    "\n",
    "- 使学习快速进行（能够使用较大的学习率）\n",
    "  \n",
    "- 降低模型对初始值的敏感性\n",
    "  \n",
    "- 从一定程度上抑制过拟合\n",
    "\n",
    "BatchNorm**主要思路**\n",
    "\n",
    "是在训练时按**mini-batch**为单位，对神经元的数值进行归一化，使数据的分布满足均值为0，方差为1。\n",
    "\n",
    "具体计算过程如下：\n",
    "\n",
    "**1. 计算mini-batch内样本的均值**\n",
    "\n",
    "**2. 计算mini-batch内样本的方差**\n",
    "\n",
    "**3. 计算标准化之后的输出**\n",
    "\n",
    "- 输入 x, [N, K]\n",
    "- 输出 y, [N, K]\n",
    "- 均值 $\\mu_B$，[K, ]\n",
    "- 方差 $\\sigma_B^2$, [K, ]\n",
    "- 缩放参数$\\gamma$, [K, ]\n",
    "- 平移参数$\\beta$, [K, ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output of BatchNorm Layer: \n",
      " [[-1.2247438 -1.2247438 -1.2247438]\n",
      " [ 0.         0.         0.       ]\n",
      " [ 1.2247438  1.2247438  1.2247438]]\n",
      "std 4.0, mean 2.449489742783178, \n",
      " output [-1.22474487  0.          1.22474487]\n"
     ]
    }
   ],
   "source": [
    "# 输入数据形状是 [N, K]时的示例\n",
    "import numpy as np\n",
    "\n",
    "import paddle\n",
    "import paddle.fluid as fluid\n",
    "from paddle.fluid.dygraph.nn import BatchNorm\n",
    "# 创建数据\n",
    "data = np.array([[1,2,3], [4,5,6], [7,8,9]]).astype('float32')\n",
    "# 使用BatchNorm计算归一化的输出\n",
    "with fluid.dygraph.guard():\n",
    "    # 输入数据维度[N, K]，num_channels等于K\n",
    "    bn = BatchNorm(num_channels=3)    \n",
    "    x = fluid.dygraph.to_variable(data)\n",
    "    y = bn(x)\n",
    "    print('output of BatchNorm Layer: \\n {}'.format(y.numpy()))\n",
    "\n",
    "# 使用Numpy计算均值、方差和归一化的输出\n",
    "# 这里对第0个特征进行验证\n",
    "a = np.array([1,4,7])\n",
    "a_mean = a.mean()\n",
    "a_std = a.std()\n",
    "b = (a - a_mean) / a_std\n",
    "print('std {}, mean {}, \\n output {}'.format(a_mean, a_std, b))\n",
    "\n",
    "# 建议读者对第1和第2个特征进行验证，观察numpy计算结果与paddle计算结果是否一致"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **示例二：** 当输入数据形状是$[N, C, H, W]$时， 一般对应卷积层的输出，示例代码如下所示。\n",
    "\n",
    "这种情况下会沿着C这一维度进行展开，分别对每一个通道计算N个样本中总共$N\\times H \\times W$个像素点的均值和方差，数据和参数对应如下：\n",
    "- 输入 x, [N, C, H, W]\n",
    "- 输出 y, [N, C, H, W]\n",
    "- 均值 $\\mu_B$，[C, ]\n",
    "- 方差 $\\sigma_B^2$, [C, ]\n",
    "- 缩放参数$\\gamma$, [C, ]\n",
    "- 平移参数$\\beta$, [C, ]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input of BatchNorm Layer: \n",
      " [[[[0.54340494 0.2783694  0.4245176 ]\n",
      "   [0.84477615 0.00471886 0.12156912]\n",
      "   [0.67074907 0.82585275 0.13670659]]\n",
      "\n",
      "  [[0.5750933  0.89132196 0.20920213]\n",
      "   [0.18532822 0.10837689 0.21969749]\n",
      "   [0.9786238  0.8116832  0.17194101]]\n",
      "\n",
      "  [[0.81622475 0.27407375 0.4317042 ]\n",
      "   [0.9400298  0.81764936 0.33611196]\n",
      "   [0.17541045 0.37283206 0.00568851]]]\n",
      "\n",
      "\n",
      " [[[0.25242636 0.7956625  0.01525497]\n",
      "   [0.5988434  0.6038045  0.10514768]\n",
      "   [0.38194343 0.03647606 0.89041156]]\n",
      "\n",
      "  [[0.98092085 0.05994199 0.89054596]\n",
      "   [0.5769015  0.7424797  0.63018394]\n",
      "   [0.5818422  0.02043913 0.21002658]]\n",
      "\n",
      "  [[0.5446849  0.76911515 0.25069523]\n",
      "   [0.2858957  0.8523951  0.9750065 ]\n",
      "   [0.8848533  0.35950786 0.59885895]]]]\n",
      "output of BatchNorm Layer: \n",
      " [[[[ 0.41260773 -0.46198368  0.02029113]\n",
      "   [ 1.4071033  -1.3650038  -0.9794093 ]\n",
      "   [ 0.83283097  1.344658   -0.9294571 ]]\n",
      "\n",
      "  [[ 0.25201762  1.2038352  -0.8492796 ]\n",
      "   [-0.92113775 -1.1527538  -0.81768954]\n",
      "   [ 1.4666054   0.9641302  -0.9614319 ]]\n",
      "\n",
      "  [[ 0.9541145  -0.9075854  -0.366296  ]\n",
      "   [ 1.3792504   0.9590065  -0.69455147]\n",
      "   [-1.2463866  -0.56845784 -1.8291972 ]]]\n",
      "\n",
      "\n",
      " [[[-0.5475932   1.2450331  -1.3302356 ]\n",
      "   [ 0.5955492   0.6119205  -1.0335984 ]\n",
      "   [-0.12019946 -1.2602081   1.5576957 ]]\n",
      "\n",
      "  [[ 1.4735192  -1.2985382   1.2014996 ]\n",
      "   [ 0.25746003  0.75583434  0.41783503]\n",
      "   [ 0.272331   -1.4174379  -0.84679806]]\n",
      "\n",
      "  [[ 0.02166999  0.7923442  -0.9878652 ]\n",
      "   [-0.8669898   1.0783204   1.4993575 ]\n",
      "   [ 1.189779   -0.614212    0.20769906]]]]\n",
      "channel 0 of input data: \n",
      " [[[0.54340494 0.2783694  0.4245176 ]\n",
      "  [0.84477615 0.00471886 0.12156912]\n",
      "  [0.67074907 0.82585275 0.13670659]]\n",
      "\n",
      " [[0.25242636 0.7956625  0.01525497]\n",
      "  [0.5988434  0.6038045  0.10514768]\n",
      "  [0.38194343 0.03647606 0.89041156]]]\n",
      "std 0.4183686077594757, mean 0.3030227720737457, \n",
      " output: \n",
      " [[[ 0.41263014 -0.46200886  0.02029219]\n",
      "  [ 1.4071798  -1.3650781  -0.9794626 ]\n",
      "  [ 0.8328762   1.3447311  -0.92950773]]\n",
      "\n",
      " [[-0.54762304  1.2451009  -1.3303081 ]\n",
      "  [ 0.5955816   0.61195374 -1.0336547 ]\n",
      "  [-0.12020606 -1.2602768   1.5577804 ]]]\n"
     ]
    }
   ],
   "source": [
    "# 输入数据形状是[N, C, H, W]时的batchnorm示例\n",
    "import numpy as np\n",
    "\n",
    "import paddle\n",
    "import paddle.fluid as fluid\n",
    "from paddle.fluid.dygraph.nn import BatchNorm\n",
    "\n",
    "# 设置随机数种子，这样可以保证每次运行结果一致\n",
    "np.random.seed(100)\n",
    "# 创建数据\n",
    "data = np.random.rand(2,3,3,3).astype('float32')\n",
    "# 使用BatchNorm计算归一化的输出\n",
    "with fluid.dygraph.guard():\n",
    "    # 输入数据维度[N, C, H, W]，num_channels等于C\n",
    "    bn = BatchNorm(num_channels=3)\n",
    "    x = fluid.dygraph.to_variable(data)\n",
    "    y = bn(x)\n",
    "    print('input of BatchNorm Layer: \\n {}'.format(x.numpy()))\n",
    "    print('output of BatchNorm Layer: \\n {}'.format(y.numpy()))\n",
    "\n",
    "# 取出data中第0通道的数据，\n",
    "# 使用numpy计算均值、方差及归一化的输出\n",
    "a = data[:, 0, :, :]\n",
    "a_mean = a.mean()\n",
    "a_std = a.std()\n",
    "b = (a - a_mean) / a_std\n",
    "print('channel 0 of input data: \\n {}'.format(a))\n",
    "print('std {}, mean {}, \\n output: \\n {}'.format(a_mean, a_std, b))\n",
    "\n",
    "# 提示：这里通过numpy计算出来的输出\n",
    "# 与BatchNorm算子的结果略有差别，\n",
    "# 因为在BatchNorm算子为了保证数值的稳定性，\n",
    "# 在分母里面加上了一个比较小的浮点数epsilon=1e-05"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 丢弃法（Dropout）\n",
    "\n",
    "丢弃法（Dropout）是深度学习中一种常用的抑制过拟合的方法\n",
    "\n",
    "1.神经网络学习过程中，随机删除一部分神经元。训练时，随机选出一部分神经元，将其输出设置为0，这些神经元将不对外传递信号。\n",
    "\n",
    "\n",
    "在预测场景时，会向前传递所有神经元的信号，可能会引出一个新的问题：训练时由于部分神经元被随机丢弃了，输出数据的总大小会变小。比如：计算其$L1$范数会比不使用Dropout时变小，但是预测时却没有丢弃神经元，这将导致训练和预测时数据的分布不一样。为了解决这个问题，飞桨支持如下两种方法：\n",
    "\n",
    "- **downgrade_in_infer**\n",
    "\n",
    "训练时以比例$r$随机丢弃一部分神经元，不向后传递它们的信号；预测时向后传递所有神经元的信号，但是将每个神经元上的数值乘以 $(1 - r)$。\n",
    "\n",
    "- **upscale_in_train**\n",
    "\n",
    "训练时以比例$r$随机丢弃一部分神经元，不向后传递它们的信号，但是将那些被保留的神经元上的数值除以 $(1 - r)$；预测时向后传递所有神经元的信号，不做任何处理。\n",
    "\n",
    "在飞桨[dropout API](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/layers_cn/dropout_cn.html#dropout)中，paddle.fluid.layers.dropout通过dropout_implementation参数来指定用哪种方式对神经元进行操作，dropout_implementation参数的可选值是'downgrade_in_infer'或'upscale_in_train'，缺省值是'downgrade_in_infer'。\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "不同框架对于dropout的默认处理方式可能不同\n",
    "\n",
    "------\n",
    "\n",
    "飞桨[dropout API](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/layers_cn/dropout_cn.html#dropout)包含的主要参数如下：\n",
    "\n",
    "- x （Variable）- 数据类型是Tensor，需要采用丢弃法进行操作的对象。\n",
    "- dropout_prob (float32) - 对$x$中元素进行丢弃的概率，即输入单元设置为0的概率，该参数对元素的丢弃概率是针对于每一个元素而言而不是对所有的元素而言。举例说，假设矩阵内有12个数字，经过概率为0.5的dropout未必一定有6个零。\n",
    "- is_test (bool) - 是否运行在测试阶段，由于dropout在训练和测试阶段表现不一样，通过此参数控制其表现，默认值为'False'。\n",
    "- dropout_implementation (str) - 丢弃法的实现方式，有'downgrade_in_infer'和'upscale_in_train'两种，具体情况请见上面的说明，默认是'downgrade_in_infer'。\n",
    "\n",
    "下面这段程序展示了经过dropout之后输出数据的形式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x1 [[[[0.54340494 0.2783694  0.4245176 ]\n",
      "   [0.84477615 0.00471886 0.12156912]\n",
      "   [0.67074907 0.82585275 0.13670659]]\n",
      "\n",
      "  [[0.5750933  0.89132196 0.20920213]\n",
      "   [0.18532822 0.10837689 0.21969749]\n",
      "   [0.9786238  0.8116832  0.17194101]]\n",
      "\n",
      "  [[0.81622475 0.27407375 0.4317042 ]\n",
      "   [0.9400298  0.81764936 0.33611196]\n",
      "   [0.17541045 0.37283206 0.00568851]]]\n",
      "\n",
      "\n",
      " [[[0.25242636 0.7956625  0.01525497]\n",
      "   [0.5988434  0.6038045  0.10514768]\n",
      "   [0.38194343 0.03647606 0.89041156]]\n",
      "\n",
      "  [[0.98092085 0.05994199 0.89054596]\n",
      "   [0.5769015  0.7424797  0.63018394]\n",
      "   [0.5818422  0.02043913 0.21002658]]\n",
      "\n",
      "  [[0.5446849  0.76911515 0.25069523]\n",
      "   [0.2858957  0.8523951  0.9750065 ]\n",
      "   [0.8848533  0.35950786 0.59885895]]]], \n",
      " out1_1 \n",
      " [[[[0.         0.         0.        ]\n",
      "   [0.84477615 0.         0.12156912]\n",
      "   [0.         0.         0.13670659]]\n",
      "\n",
      "  [[0.         0.89132196 0.20920213]\n",
      "   [0.         0.10837689 0.        ]\n",
      "   [0.         0.8116832  0.        ]]\n",
      "\n",
      "  [[0.         0.         0.        ]\n",
      "   [0.         0.81764936 0.33611196]\n",
      "   [0.17541045 0.37283206 0.        ]]]\n",
      "\n",
      "\n",
      " [[[0.         0.7956625  0.01525497]\n",
      "   [0.5988434  0.         0.        ]\n",
      "   [0.         0.         0.        ]]\n",
      "\n",
      "  [[0.         0.         0.        ]\n",
      "   [0.5769015  0.         0.63018394]\n",
      "   [0.         0.         0.21002658]]\n",
      "\n",
      "  [[0.5446849  0.         0.        ]\n",
      "   [0.2858957  0.8523951  0.        ]\n",
      "   [0.8848533  0.35950786 0.        ]]]], \n",
      " out1_2 \n",
      " [[[[0.54340494 0.         0.        ]\n",
      "   [0.84477615 0.00471886 0.12156912]\n",
      "   [0.67074907 0.82585275 0.        ]]\n",
      "\n",
      "  [[0.         0.89132196 0.        ]\n",
      "   [0.18532822 0.         0.        ]\n",
      "   [0.9786238  0.8116832  0.17194101]]\n",
      "\n",
      "  [[0.81622475 0.         0.        ]\n",
      "   [0.9400298  0.         0.        ]\n",
      "   [0.17541045 0.         0.        ]]]\n",
      "\n",
      "\n",
      " [[[0.25242636 0.         0.        ]\n",
      "   [0.5988434  0.6038045  0.10514768]\n",
      "   [0.38194343 0.         0.        ]]\n",
      "\n",
      "  [[0.         0.         0.89054596]\n",
      "   [0.5769015  0.7424797  0.        ]\n",
      "   [0.5818422  0.         0.21002658]]\n",
      "\n",
      "  [[0.5446849  0.         0.25069523]\n",
      "   [0.2858957  0.         0.9750065 ]\n",
      "   [0.8848533  0.         0.59885895]]]]\n",
      "x2 [[ 1.  2.  3.]\n",
      " [ 4.  5.  6.]\n",
      " [ 7.  8.  9.]\n",
      " [10. 11. 12.]], \n",
      " out2_1 \n",
      " [[ 2.  0.  6.]\n",
      " [ 8. 10.  0.]\n",
      " [14. 16. 18.]\n",
      " [ 0.  0. 24.]], \n",
      " out2_2 \n",
      " [[ 0.  0.  6.]\n",
      " [ 0. 10. 12.]\n",
      " [ 0. 16. 18.]\n",
      " [ 0. 22.  0.]]\n"
     ]
    }
   ],
   "source": [
    "# dropout操作\n",
    "import numpy as np\n",
    "\n",
    "import paddle\n",
    "import paddle.fluid as fluid\n",
    "\n",
    "# 设置随机数种子，这样可以保证每次运行结果一致\n",
    "np.random.seed(100)\n",
    "# 创建数据[N, C, H, W]，一般对应卷积层的输出\n",
    "data1 = np.random.rand(2,3,3,3).astype('float32')\n",
    "# 创建数据[N, K]，一般对应全连接层的输出\n",
    "data2 = np.arange(1,13).reshape([-1, 3]).astype('float32')\n",
    "# 使用dropout作用在输入数据上\n",
    "with fluid.dygraph.guard():\n",
    "    x1 = fluid.dygraph.to_variable(data1)\n",
    "    out1_1 = fluid.layers.dropout(x1, dropout_prob=0.5, is_test=False)\n",
    "    out1_2 = fluid.layers.dropout(x1, dropout_prob=0.5, is_test=True)\n",
    "\n",
    "    x2 = fluid.dygraph.to_variable(data2)\n",
    "    out2_1 = fluid.layers.dropout(x2, dropout_prob=0.5, \\\n",
    "                    dropout_implementation='upscale_in_train')\n",
    "    out2_2 = fluid.layers.dropout(x2, dropout_prob=0.5, \\\n",
    "                    dropout_implementation='upscale_in_train', is_test=True)\n",
    "    \n",
    "    print('x1 {}, \\n out1_1 \\n {}, \\n out1_2 \\n {}'.format(data1, out1_1.numpy(),  out1_2.numpy()))\n",
    "    print('x2 {}, \\n out2_1 \\n {}, \\n out2_2 \\n {}'.format(data2, out2_1.numpy(),  out2_2.numpy()))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
